kvm: fill the template's holes before taking its LUKS clone snapshot - #14

Implements encrypted RBD data and root disks using librbd's native LUKS2
encryption, decrypted at runtime by libvirt/qemu via <encryption
engine='librbd'>. CloudStack manages the passphrase (existing model).

- RbdEncryption: isolated helper wrapping `rbd encryption format luks2`,
  cephx via --id + keyfile (secret not on the command line), LUKS passphrase
  via KeyFile. Kept separate so the CLI can later be swapped for a JNA binding
  (rados-java has no rbd_encryption_format API).
- LibvirtStorageAdaptor: create/clone the raw RBD image, then apply
  `rbd encryption format luks2`; mark the disk LUKS2 so encrypt_format
  propagates to the volume. Replaces the fail-closed guards.
- QemuObject.EncryptFormat: add LUKS2.
- LibvirtVMDef: render <encryption format='luks2' engine='librbd'>; the
  encrypt details now carry an optional engine.
- attach (KVMStorageProcessor) and boot (LibvirtComputingResource): set
  engine='librbd' for RBD-backed encrypted volumes.

NOTE: the CoW-clone-then-format path (encrypted root from an unencrypted
template) needs live-cluster validation for the parent-grow / usable-size
behaviour described in the Ceph image-encryption docs.

Builds: api + plugins/hypervisors/kvm (JDK11).

hostSupportsVolumeEncryption() now advertises encryption capability if the
host supports EITHER qemu-native LUKS (qemu-img LUKS + cryptsetup) OR librbd
native encryption (rbd CLI with the encryption subcommand). Previously a
Ceph-only host that lacked cryptsetup would not advertise encryption even
though librbd can encrypt RBD volumes.

Split into hostSupportsQemuNativeVolumeEncryption() and
hostSupportsRbdVolumeEncryption(); kept HOST_VOLUME_ENCRYPTION as the single
host-wide flag (documented limitation: not per-pool).

Encrypted RBD volumes are encrypted natively by librbd and must be resized
with `rbd resize --encryption-passphrase-file` so librbd grows the encrypted
payload and keeps the LUKS header consistent. The existing encrypted-resize
path (resizeEncryptedQcowFile) uses qemu-img --object secret, which is for
qemu-native LUKS and does not fit the librbd LUKS2 layout.

- RbdEncryption.resize(): new `rbd resize` wrapper (cephx via --id + keyfile,
  passphrase via KeyFile, optional --allow-shrink).
- LibvirtResizeVolumeCommandWrapper: detect encrypted RBD and route to the rbd
  resize path, bypassing the libvirt v.resize and qemu-img paths.

Snapshot/revert, RBD<->RBD copy, and migration of encrypted RBD volumes need
no code changes: they operate on the raw (LUKS-containing) image at the block
level, and the destination passphrase secret is already created engine-agnostic
in LibvirtPrepareForMigrationCommandWrapper. These still require live validation.

Builds: plugins/hypervisors/kvm (JDK11).

For a running VM, an librbd-encrypted RBD volume must be resized in-band by
qemu/librbd, not out-of-band by the rbd CLI. Gate the CLI rbd-resize path on
!vmIsRunning so:
 - offline -> `rbd resize --encryption-passphrase-file` (librbd-aware), and
 - online  -> existing NOTIFYONLY path -> virsh blockresize, where qemu's
   block_resize delegates to librbd to grow the encrypted payload and notify
   the guest in one step (no passphrase needed; qemu holds the secret).

This avoids notify-less out-of-band growth and qemu/librbd size divergence
while the image is open. Online behaviour still needs live validation that
blockresize resizes the encrypted payload for engine='librbd' disks.

Root disks could not be encrypted: cloning a plaintext template and then
`rbd encryption format`ing the clone leaves the inherited OS data unreadable
(the LUKS header offsets it), so the guest could not mount root.

Fix, in createDiskFromTemplateOnRBD, with two paths:
- Option A (same-cluster cached RBD template): grow the template base to
  reserve LUKS2 header space, snapshot+protect it (cloudstack-base-snap-luks),
  clone from it, apply the LUKS2 header, resize the clone to the requested
  size. Inherited template data stays readable through the clone's encryption
  and the clone is a thin CoW image (only the header is written).
- Option B (first-use / non-RBD template): create an empty image, apply a
  LUKS2 header, then import the template THROUGH the encryption layer via
  RbdEncryption.importTemplate (qemu-img convert -n into encrypt.key-secret).
  Correct but a full copy.

Validated end-to-end on Ubuntu 26.04 / libvirt 12.0.0: both boot; A is thin
(3.5 GiB provisioned, ~120 MiB used); LUKS2 verified at rest on Ceph.

Review pass over the librbd LUKS2 encryption feature to fix latent issues
and bring it in line with CloudStack conventions:

- RbdEncryption: reject empty/null passphrase with a clear error; round
  rbd --size up to MiB so a non-aligned request never shrinks the volume
  below what was asked for; create the temporary cephx conf/keyring 0600
  explicitly instead of relying on the umask.
- LibvirtStorageAdaptor: close Rados/IoCTX/RbdImage in a finally block on
  the encrypted-root paths (mirrors deleteVolume) so handles are not
  leaked on exceptions; use parameterized log messages instead of string
  concatenation; extract the encrypted-root Option A/B logic into
  createEncryptedRootCoWClone / createEncryptedRootFullCopy.
- RbdEncryption: use an instance logger (matching the plugin convention)
  and split argv construction into build{Format,Resize,Convert}Script so
  the generated commands can be unit-tested.

Assert the rbd/qemu-img argv built for format, resize and
convert-through-encryption (RBD and file sources), and that empty/null
passphrases are rejected. Command construction is verified without a
live Ceph cluster.

libvirt 10.0.0 has an object apply-order bug (fixed in 10.1.0) that breaks
hot-plug of an encrypted rbd blockdev: on attach the disk is opened before its
LUKS secret object is defined, so the attach fails with "No secret with id
'...-format-encryption-secret0'". Booting a VM from an encrypted RBD disk is
unaffected (the QEMU command line resolves all -object before -blockdev).

Refuse the attach up front with a clear error (mirroring the existing
openvswitch/io_uring libvirt-version gates) instead of letting libvirt fail
opaquely. Only the RBD hot-plug path is gated; boot/root/detach are untouched.

RbdEncryption built its own 'qemu-img convert' command line, which
duplicated qemu-img knowledge outside of QemuImg. QemuImg could only
write to a plain filename destination, so importing a template through
the librbd encryption layer was not expressible with it.

QemuImg now supports a destination described by image options
(--target-image-opts, with -n implied since such a target always exists
already), exposed as convertIntoExistingTarget(). QemuImageOptions can
render its parameters under either image-opts flag.

RbdEncryption.importTemplate now composes QemuImageOptions and a
QemuObject secret and delegates to QemuImg; its hand-built convert
script is removed. The rbd CLI calls (encryption format, resize,
support probe) stay, as qemu-img cannot perform them.

No functional change to the generated command.

Review feedback: single-letter and abbreviated names are hard to read.
Renamed in the two methods added by this PR only (renaming the rest of
the class is out of scope here): r -> radosConnection, io -> ioContext,
rbd -> rbdClient, base -> templateImage, s -> snapshotInfo, encSnap ->
luksReservedSnapshotName, haveEncSnap -> luksSnapshotExists, createSize
-> imageSizeWithLuksHeader, srcIsRbd -> sourceIsRbdPool.

No functional change.

Review feedback: distinguish the two volume encryption mechanisms
instead of advertising them under one host flag.

host.volume.encryption goes back to meaning qemu-native LUKS only
(qemu-img LUKS + cryptsetup), as it did before this PR, and hosts now
additionally report host.volume.encryption.rbd for librbd encryption
(rbd encryption format).

The deployment planner requires the flag matching the pool type of each
encrypted volume - librbd for volumes on RBD pools, qemu-native for any
other pool type - at all three places it validated encryption support
before. The pool is taken from the pools proposed alongside the host
when present, so first deployments are matched accurately too; an
encrypted volume with no pool yet accepts either mechanism and the
storage pool allocator picks a pool the host can serve.

This also stops a host whose librbd is too old for 'rbd encryption
format' from being selected for encrypted RBD volumes; it previously
advertised encryption through the qemu stack and the VM failed to
start.

takeRbdVolumeSnapshotOfStoppedVm() called image.snapCreate(snapshotName)
twice. The first call creates the RBD snapshot, the second one always
throws RbdException ("Failed to create snapshot <uuid>") because the
snapshot already exists.

The duplicate is a merge artifact: 30d3066 ("Merge branch '4.20' into
4.22") resolved a conflict by keeping the call from both sides - each
parent had exactly one.

Because there was no finally block, that exception skipped rbd.close(image)
and r.ioCtxDestroy(io), so the agent kept the image open and held its RBD
exclusive-lock indefinitely. The exception is only logged, so the snapshot
job still reported success and the fault stayed invisible.

Consequences observed on a KVM + Ceph/RBD cluster:

- revertSnapshot fails with "com.ceph.rbd.RbdException: Failed to rollback
  snapshot <uuid>". librbd returns EROFS because a live peer holds the
  exclusive-lock; 'rbd snap rollback' only succeeds once that client dies
  and librbd can break the lock, which makes the failure look intermittent.
- getRbdSnapshotSize() is never reached, so every snapshot is reported with
  physical size 0 when snapshot.backup.to.secondary is false.
- The leaked watchers keep the image busy, so 'rbd rm' fails and the volume
  cannot be expunged - it stays stuck in state Destroy.

Note the method also runs for RUNNING VMs: createSnapshot() branches on
"RUNNING && !primaryPool.isExternalSnapshot()", and RBD is an
external-snapshot pool, so every RBD volume snapshot took this path.

Remove the duplicated call and move the image/IO-context cleanup into a
finally block so the lock is released even if the snapshot itself fails.

…napshot

createRBDvolumeFromRBDSnapshot() closed the source image, the cloned image
and the RADOS IO context only on the success path, and called snapUnprotect()
only there too. Two paths escaped that cleanup:

- the early "Could not find snapshot ... on RBD" return, and
- any RadosException/RbdException from clone(), resize() or flatten(), which
  is caught and turned into a null disk.

Both leave the images open, so this client keeps the RBD exclusive-lock. That
later makes 'rbd snap rollback' (revertSnapshot) fail with EROFS from another
host, and keeps the image busy so 'rbd rm' cannot remove it - the volume then
stays stuck in state Destroy.

The failure paths after snapProtect() are worse: the snapshot stays protected,
and a protected snapshot can be deleted neither on its own nor together with
its volume.

Move the cleanup into a finally block, tracking whether the snapshot was
actually protected so it is unprotected exactly when it needs to be. Failures
during cleanup are logged and never mask the original outcome; a failed
snapUnprotect is logged at ERROR since it needs manual intervention.

This is the same class of defect as the leak fixed in
takeRbdVolumeSnapshotOfStoppedVm(); no behaviour changes on the success path.

Two tests around takeRbdVolumeSnapshotOfStoppedVm, using the MockedConstruction
pattern already used in this test class (the Rbd instance is created inside the
method under test, so it cannot be injected):

- createsSnapshotExactlyOnce guards the duplicated snapCreate call from coming
  back, and checks the image and IO context are released.
- releasesHandlesWhenSnapshotFails makes snapCreate throw and asserts the image
  is still closed and the IO context destroyed, so a future failure cannot leak
  the RBD exclusive-lock again.

takeRbdVolumeSnapshotOfStoppedVm, radosConnect and getRbdSnapshotSize widened
from private to protected so the test can stub the Ceph interactions.

Address review feedback: the nested try-catch teardown blocks in the
finally clauses of takeRbdVolumeSnapshotOfStoppedVm and
createRBDvolumeFromRBDSnapshot are extracted into two reusable,
null-safe helpers that log but never throw:

- closeRbdImage (3 call sites)
- destroyRadosIoCtx (2 call sites)

No behavior change.

Follow-up to the teardown helpers: the snapUnprotect block in the
finally clause of createRBDvolumeFromRBDSnapshot moves into a
never-throwing unprotectRbdSnapshot helper, so the finally clause is
now free of inline try-catch constructions entirely.

No behavior change.

An in-place restore of a KVM instance with more than one data disk can write
a backed up disk into a different volume than the one it came from.

NASBackupProvider.restoreVMBackup pairs the backed up volumes with the
instance's current volumes by position in two lists that are each sorted by
device id, and LibvirtRestoreBackupCommandWrapper.restoreVolumesOfExistingVM
consumes those lists index by index. The volume UUID is carried along but only
used to build error messages.

That pairing is only correct while the instance's device ids still match the
ones recorded in the backup, and the restore itself breaks that: on completion
KVMGuru.importVirtualMachineFromBackup re-attaches every data disk with
getNextAvailableDeviceId(). The volumes are never detached first, so their
current ids still count as in use and the helper cannot return the id a volume
already holds - every data disk is shifted by one slot on each restore. The
root disk is unaffected because it is always re-attached at device id 0.

A second restore therefore maps the backup files onto the wrong volumes. With
equally sized disks the contents are silently exchanged and the API reports
success. With disks of different sizes the first volume is overwritten and the
restore then fails converting an image into a smaller volume, leaving that
volume half rewritten with no rollback.

Three changes:

- KVMGuru: re-attach data disks with the device id recorded in the backup,
  falling back to the next free id only when the backup has none.
- LibvirtRestoreBackupCommandWrapper: select the target volume by UUID instead
  of by list position, and fail explicitly when a backed up volume is no longer
  attached to the instance.
- NASBackupProvider: derive the UUID list and the backup file list from the
  same sorted collection so the two cannot drift apart.

The create-instance-from-backup path (restoreVolumesOfDestroyedVMs) is left
alone: it provisions new volumes with new UUIDs, so it has nothing to match on.

LibvirtRestoreBackupCommandWrapperTest: restoring an instance whose volumes
arrive in a different order than the backed up volumes must still write each
backup into the volume it was taken from, and must fail explicitly when a
backed up volume is no longer attached to the instance.

KVMGuruTest: importing an instance from a backup must re-attach its data disks
on the device ids recorded in the backup rather than allocating new ones.

… uuid

Matching backed up volumes to the instance's volumes by uuid breaks
creating an instance from a backup: that instance gets brand new volumes,
so none of the uuids recorded in the backup can ever match and the
restore always failed with the volume 'is not attached to the instance
any more' error.

When none of the recorded uuids match, the volumes are new and the device
id ordering both lists already carry is the only mapping available, so
fall back to it. When some of them do match, a missing one really is a
detached volume and is still rejected, which is what the uuid matching was
added for. The fallback also refuses to run if the two lists differ in
length.

Two defects in one line, both fatal for a stopped-instance backup:

    mount ... 2>&1 | tee -a "$logFile"
    if [ $? -eq 0 ]; then

1. `tee` writes to the log AND to stdout, so anything the mount helper says
   lands in the stream the agent parses. `LibvirtTakeBackupCommandWrapper
   .parseBackupSize` sums EVERY line of that output as a number when volume
   paths are set - which is only for a STOPPED instance - so a single benign
   warning throws NumberFormatException. A Ceph client reporting a missing
   keyring (harmless: the mount succeeds) is enough. The record then sits in
   BackingUp for ever: there is no timeout, no transition to Failed, and it
   blocks every later restore of that instance.

   Measured on 4.23.0.0-SNAPSHOT with Ceph primary storage: without the
   keyring the backup wedges with
   `NumberFormatException: For input string: "2026-08-25T17:38:38.403+0000"`
   - the timestamp of the mount warning. A running instance is unaffected,
   because with no volume paths the parser reads only the last line.

2. `$?` after a pipeline is the exit status of the LAST command, i.e. `tee`,
   which practically never fails. A mount that did fail was therefore
   reported as "Successfully mounted" and the backup carried on against an
   empty directory.

Same change as the pending upstream PRs apache#12843 and apache#12898.

parseBackupSize() parsed every line of the backup script's stdout as a byte
count. The script is not the only writer on that stream: mount helpers and
storage clients emit warnings of their own, and one such line was enough to
throw NumberFormatException out of the wrapper, e.g.

  java.lang.NumberFormatException: For input string: "2026-08-25T17:38:38.403+0000"

emitted by a Ceph client as "unable to find a keyring on /etc/ceph/ceph.keyring".

The cost of that is out of proportion to the cause. The exception escapes after
the backup has already been written, so the data is on the repository but the
record stays in BackingUp - there is no timeout and no transition to Failed -
and it blocks every later restore of that instance until the row is corrected
by hand.

Ignore lines that do not begin with a byte count instead, logging them at debug.
Only the multi-volume branch was reachable in practice, which is the branch
taken for a stopped instance, so running instances backed up normally and the
failure went unnoticed.

Adds LibvirtTakeBackupCommandWrapperTest; four of its seven cases fail without
this change.

An encrypted volume was backed up in the clear: libvirt's backup-begin
copies from the top of the disk's block graph, which is the decrypted
guest view, and the backup target carried no encryption. Restore then
wrote that plaintext back over a volume whose record still says luks,
leaving an image the hypervisor could not open. This affected every
encrypted volume, not just RBD ones, since the leak is independent of
which engine decrypts the source.

QEMU now writes the backup already LUKS-encrypted, using the volume's own
passphrase, via the encryption element libvirt allows inside the backup
target. The stopped-VM path writes the same artefact through qemu-img.
Restore reads the backup with that passphrase and writes it back through
the volume's own encryption, so the restored volume keeps its LUKS header.

The passphrases travel on the take/restore commands, are marked so they
never reach the logs, and are wiped once handled. The agent registers an
ephemeral libvirt secret per volume under a distinct consumer name, so
cleanup cannot remove the secret a running VM is using for the same disk.

Encrypted backups onto non-RBD pools are rejected explicitly rather than
silently corrupting the volume.

Makes the existing take-backup test explicit about its volumes being
unencrypted, and adds a test asserting the passphrase of an encrypted
volume is sent on the command, which is what lets the agent have QEMU
write the backup already encrypted.

Three problems only a real restore surfaced:

The pre-restore 'qemu-img check' ran without a key, so an encrypted
backup looked corrupt and the restore refused to start.

The passphrase map is keyed by the volume path as the management server
knows it, but the agent sees that path with a pool prefix, so the lookup
missed and the code silently fell through to the unencrypted path.

QemuImageOptions built from a raw map has no format set, and in that case
renders as a bare filename, dropping the backup's own encrypt.key-secret;
the source options now force the image-opts form.

The backport of the restore-volume-mapping regression tests carried an
unused-import cleanup that was correct on the branch it was written against
but not on 4.23.0.0, where ArrayList and Arrays are used by the backup
repository mount handling. EnumMap is needed by restoreEncryptedBlockDevice.

LibvirtRestoreBackupCommandWrapperTest: the restore path in 4.23 first asks
qemu-img whether the backup references a parent and flattens the chain with
qemu-img convert when it does; only a backup with no backing file is rsynced.
The tests carried over from the 4.22 branch stubbed
runSimpleBashScriptForExitValue to 0 for every command, which made the probe
report a backing chain and take the convert path, so the rsync argv they
assert on was never executed.

NASBackupProviderTest: drop an ArgumentCaptor import duplicated when the
encrypted-volume test was merged onto 4.23's import block, and pass the
isolated flag that takeBackup gained in 4.23.

An encrypted root is a librbd CoW clone of a plaintext template carrying a
LUKS2 header. librbd serves the ranges the clone's own objects do not hold
from that parent, as plaintext, which is what makes the inherited filesystem
readable. That stops the moment an object exists in the clone: one 4 KiB guest
write copies up the whole 4 MiB object, and the ranges inside it that the
parent never materialised are from then on read through the crypto layer -
where zeros decrypted with AES-XTS are not zeros.

Before taking the snapshot encrypted roots are cloned from, rewrite the
template image from its own cloudstack-base-snap with qemu-img convert -S 0,
so the snapshot has no holes left. The snapshot gets a new name, -luks2, so
templates prepared before get a dense one too; roots already cloned from the
old -luks snapshot keep it. If another host prepares the same template at the
same time, its protected snapshot is used instead of failing the deploy.

QemuImg grows setWriteZeroRanges() for the -S 0 this needs, since qemu-img
skips the source's zero ranges by default.